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(54) Information device and computer product 

(57) A transmission-side cellular phone (20) sends, 
via a mail server (40), to a reception -side cellular phone 
(30), a face image and its attribute information as an 
attachment to an email. The reception-side cellular 
phone (30) extracts the face image from the email, se- 
lects face part images from among a plurality of face 



part images stored in the face part storing section, com- 
bines the face image and the selected face parts, and 
displays the combined image. The reception -side cellu- 
lar phone (30) selects the face part images based on, 
for example, a number of times a telephone call from a 
person was not attended in a predetermined period. 
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Description 

[0001] The present invention relates to an information 
device such as a cellular phone having a function for 
combining images. 

[0002] Portable telephone (hereinafter "cellular 
phone") with built-in camera has appeared in the mar- 
ket. This cellular phone has a possibility of being able 
to realize advanced communications that the cellular 
phones without the built-in camera have not been able 
to realize. As a result, it is expected that the cellular 
phones with built-in camera will become more popular 
and widely used. 

[0003] The cellular phone with a built-in camera has 
a function fortransmitting an image, for example, a pho- 
tograph of a face of a person, to another cellular phone 
as an attachment to an email. Some cellular phones also 
have a function to process the image, for example, to 
compresses or trim the image. The original image or the 
processes image may be used as an idle image (i.e., an 
image displayed on the idle screen). 
[0004] When an image of face of one person is trans- 
mitted to the other person, it is possible to convey the 
face or expressions on the face of the one person to the 
other person, and realize better communications be- 
tween the users. However, since the data quantity of the 
image handled by the cellular phones is high, usually 
about 8 KB, it exerts a great burden on the communica- 
tion lines, and sometimes real-timeness of the commu- 
nications is lost. 

[0005] Moreover, as the data quantity of the image ex- 
erts a great burden on the mail servers, some Internet 
providers even tend to prohibit transmission of images 
among the users. As a result, the users loose the op- 
portunity to communicate with each other using images. 
[0006] Furthermore, using of an illustration or an im- 
age of a face as the idle image was know even before 
a cellular phone with a built-in camera appeared in the 
market. In other words, the advantages of the built-in 
camera in the cellular phone have not been fully utilized. 
[0007] According to one aspect of the present inven- 
tion, a face image and a plurality of face part images 
corresponding to facial expressions are obtained; a fa- 
cial expression is determined; and the face image and 
the face part images corresponding to the facial expres- 
sion are combined to obtain a combined face image. 
The combined face image may be displayed on a dis- 
play or transmitter to other person. 
[0008] According to another aspect of the present in- 
vention, a first face image and a second face image is 
obtained; a methodforcombining is determined; and the 
first face image and the second face image are com- 
bined based on the method for combining to obtain a 
combined face image. The combined face image may 
be displayed on a display or transmitter to other person. 
[0009] According to still another aspect of the present 
invention a first face image and a second face image 
are obtained; a similarity between the first face image 



and the second face image and a relationship between 
the persons of the first face image and the second face 
image determined from the similarity are determined; 
and the similarity, the relationship or both the similarity 

5 and the relationship is displayed on a display. 

[0010] These and other objects, features and advan- 
tages of the present invention are specifically set forth 
in or will become apparent from the following detailed 
descriptions of the invention when read in conjunction 

10 with the accompanying drawings. 

Fig. 1 shows a structure of a cellular phone accord- 
ing to one embodiment of the present invention; 
Fig. 2A to Fig. 2C explain how a portrait is prepared 
15 by the cellular phone shown in Fig. 1 ; 

Fig. 3 shows one example of a system for realizing 
communications using the cellular phone shown in 
Fig. 1 ; 

Fig. 4 shows storage contents of a portrait storing 
20 section shown in Fig. 3; 

Fig. 5 shows storage contents of a face part storing 
section shown in Fig. 3; 

Fig. 6 shows storage contents of a call register stor- 
ing section shown in Fig. 3; 
25 Fig. 7 shows storage contents of an expression as- 
signment information storing section shown in Fig. 

3; 

Fig. 8 shows an example of processes executed by 
a reception-side cellular phone shown in Fig. 3; 
30 Fig. 9 shows another example of processes execut- 
ed by the reception -side cellular phone shown in 
Fig. 3; 

Fig. 10 explains how a portrait is prepared; 
Fig. 11 shows another example of a system for re- 
35 alizing communications using the cellular phone 
shown in Fig. 1 ; 

Fig. 12 shows an example of processes performed 
by a transmission-side cellular phone shown in Fig. 

11; 

40 Fig. 13 shows an example of processes performed 
by a reception-side cellular phone shown in Fig. 1 1 ; 
Fig. 1 4 shows still another example of communica- 
tions performed using the cellular phone shown in 

Fig. 1 ; 

45 Fig. 1 5 shows an example of processes performed 
by a transmission-side cellular phone shown in Fig. 
14; 

Fig. 1 6 shows still another example of communica- 
tions performed using the cellular phone shown in 

50 Fig. 1; 

Fig. 17 shows an example of processes performed 
by a transmission-side cellular phone shown in Fig. 
• 16; 

Fig. 18 shows an example of processes performed 
by a reception-side cellular phone shown in Fig. 16; 
Fig. 1 9 shows a structure of a cellular phone accord- 
ing to another embodiment of the present invention; 
Fig. 20 shows an example of processes performed 
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by a portrait combining section shown in Fig. 19; 
and 

Fig. 21 shows an example of processes performed 
by a relationship estimator shown in Fig. 19. 

[001 1] Exemplary embodiments of the cellular phone 
according to the present invention are explained in detail 
below by referring to the accompanying drawings. 
[0012] Fig. 1 shows a structure of a cellular phone 1 , 
as an example of an information device, according to 
one embodiment of the present invention. The cellular 
phone 1 comprises of a camera 10, a display 11 , a mi- 
crophone 12, a speaker 13, an image processing sec- 
tion 14, a voice processing unit 15, a communications 
section 16, and an antenna 17. 

[0013] The image processing section 14 includes a 
portrait preparing unit 14A. This portrait preparing unit 
14A receives an image (see Fig. 2A) of a face of a per- 
son (hereinafter "face image") photographed by the 
camera 1 0, converts the image into binary data, for ex- 
ample, and projects the data on a two dimensional co- 
ordinate system. Based on the projected data, the im- 
age processing section 14 determines: position of the 
overall face (hereinafter "total face), position of the eyes, 
the nose, the mouth, the hair, and the like 
(hereinafter "face parts") in the total face (see Fig, 28), 
color of the total face, and color of the hair. Finally, based 
on this information, the image processing section 14 
prepares a portrait of the face (hereinafter "portrait") 
(see Fig. 2C). When preparing the portrait, a standard 
color is used for the total face, and another standard 
color is used for the hair. The portrait of desired size may 
be prepared. It should be noted that, any method other 
than using the projected data may be used to prepare 
the portrait. 

[0014] The processed face image is then transmitted 
to, for example, other cellular phone via the communi- 
cations section 16 and the antenna 17. 
[001 5] Thus, in the cellular phone 1 of the present in- 
vention, the face image obtained by the camera 10 is 
not transmitted as it is, but the face image is processes 
to obtain a portrait and the portrait is transmitted. Since 
the portrait is used, it is possible to realize communica- 
tions between the users with a simpler and slick hard- 
ware and at an extremely high speed. 
[0016] Fig. 3 shows one example of a system for re- 
alizing communications using the cellular phone shown 
in Fig. 1 . This system comprises of a transmission-side 
cellular phone 20 and a reception-side cellular phone 
30 connected via a mail server 40. 
[0017] The transmission-side cellular phone 20 com- 
prises of a portrait preparing section 21 that prepares a 
portrait and its attribute information based on the face 
image photographed by the camera 1 0, a portrait trans- 
mitting section 22 that transmits the portrait and the at- 
tribute information to the mail server 40, and a calling 
section 23 that makes a telephone call to the reception- 
side cellular phone 30. The attribute information in- 
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eludes the position of the total face, the positions of the 
face parts, the color of the total face, and the color of 
the hair, and the like. It is preferable that, when preparing 
a portrait, the portrait preparing section 21 prepares a 
5 portrait of a predetermined size. 

[001 8] The reception-side cellular phone 30 compris- 
es of a portrait receiving section 31 that receives the 
portrait and the attribute information from the mail server 
40, a portrait storing section 32 that stores the portrait 
10 and the attribute information, a face part storing section 
33 that stores the face parts of the portrait prepared to 
change the expression of the portrait, a call register stor- 
ing section 34 that stores the record of calls received 
(hereinafter "call reception") from the transmission-side 
15 cellular phone 20, an expression assignment informa- 
tion storing section 35 that stores expression assign- 
ment information that indicates to which expression the 
expression of the portrait is to be changed, a portrait 
changing section 36 that changes the portrait stored in 
20 the portrait storing section 32, a portrait output section 
37 that outputs to the display 11 the changed portrait, 
and a call receiving section 38 that receives a call from 
the transmission-side cellular phone 20. 
[0019] The functions of sections of the transmission- 
's side cellular phone 20 and the reception-side cellular 
phone 30 may be realized by hardware or software (i. 
e., computer programs) or both. When the functions are 
realized using software, a recording medium such as a 
semiconductor memory may be provided to store the 
30 software. 

[0020] As shown in Fig. 4, the portrait storing section 

32 stores the portrait and the attribute information in cor- 
relation with the telephone number of the sender (i.e., 
the transmission-side cellular phone 20). 

35 [0021 ] As shown in Fig. 5 , the face part storing section 

33 stores expressions of the face parts in correlation 
with condition of the expression (hereinafter "expres- 
sion assignment information"). The expression assign- 
ment information is, for example, small anger, large an- 

40 ger, surprise, and the like. 

[0022] An angry face of a person may be represented 
with the eyes turned upward at the outer corners, and 
the mouth turned down at the corners. Therefore, if the 
expression assignment information is anger, then a im- 

45 ages of eyes turned upward at the outer comers, and 
image of mouth turned down at the corners are stored 
in the face part storing section 33 in correlation with the 
expression assignment information of "anger". 
[0023] The face parts stored in the face part storing 

so section 33 have sizes that match with the size of the 
portrait. The face parts with expressions may be pre- 
pared somewhere else and may be stored at the mail 
server 40. And, the transmission-side cellular phone 20 
may be made to download the face parts. 

55 [0024] The call register storing section 34 stores the 
call register information. As shown in Fig. 6, the call reg- 
ister information includes a telephone number of the 
caller, the time at which the call was received (hereinaf- 
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ter "call reception time"), and information that shows 
whether the call was attended (hereafter "absence or 
response"). 

[0025] As shown in Fig. 7, the expression assignment 
information storing section 35 stores number of times a 
call from a person was not attended (hereinafter 
"number of times of absence") and the expression as- 
signment information in correlated manner. 
[0026] A case is explained here in which expressions 
of the portrait are changed according to the number of 
times of absence using the flowcharts in Fig. 8 and Fig. 
9. The process shown in these figures is performed by 
the reception-side cellular phone 30. 
[0027] The transmission-side cellular phone 20 trans- 
mits a portrait and corresponding attribute information 
to the mail server 40 and the portrait receiving section 
31 of the reception-side cellular phone 30 receives the 
portrait and the attribute information from the mail server 
40 (step ST1). It is assumed here that the portrait is a 
face image of the user or owner of the transmission-side 
cellular phone 20. The portrait receiving section 31 of 
the reception-side cellular phone 30 also obtains the tel- 
ephone number of the transmission-side cellular phone 
20 (step ST2). Finally, the portrait, the attribute informa- 
tion, and the telephone number are stored in the portrait 
storing section 32 (step ST3). 

[0028] When a call is received from the transmission- 
side cellular phone 20, the reception-side cellular phone 
30 obtains the telephone number of the transmission- 
side cellular phone 20 (see step ST4 in Fig. 9). The por- 
trait changing section 36 extracts, from the portrait stor- 
ing section 32, the portrait and the attribute information 
relating to the telephone number obtained (step ST5). 
In other words, a portrait of the caller is obtained based 
on the telephone number. 

[0029] Next, the portrait changing section 36 obtains 
information, from the call register storing section 34, on 
how many times the call from the transmission-side cel- 
lular phone 20 was not attended during a predetermined 
period (for example, two hours) (see step ST6). Then 
the portrait changing section 36 obtains expression as- 
signment information, from the expression assignment 
information storing section 35, relating to the number of 
times the call was not attended (step ST7). 
[0030] Next, the portrait changing section 36 obtains 
face parts with expressions relating to the expression 
assignment information from the face parts storing sec- 
tion 33 (step ST8). Then the portrait changing section 
36 changes the portrait obtained via the mail server 40 
by combining the face parts with expressions with the 
portrait (step ST9). Finally, the portrait output section 37 
outputs the changed portrait to the display 1 1 where the 
portrait is displayed (step ST10). 
[0031] As a result, for example, if the telephone call 
from a caller is second time in two hours, and the owner 
of the receiver has not attended the first telephone call, 
then aface image of the callerthat is smiling is displayed 
on the display of the receiver. If the telephone call from 



a caller is third time in two hours, and the owner of the 
receiver has not attended the previous two telephone 
calls, then a face image of the caller that is little angry 
is displayed on the display of the receiver. If the tele- 
5 phone call from a caller is fourth time in two hours, and 
the owner of the receiver has not attended the previous 
three telephone calls, then, as shown in Fig. 10, aface 
image of the caller that is angry is displayed on the dis- 
play of the receiver. Thus, the receiver will know the feel- 
to ings of the caller for not attending the previous tele- 
phone calls which realizes advanced communications 
between the users. 

[0032] Fig. 1 1 shows another example of a system for 
realizing communications using the cellular phone 
15 shown in Fig. 1 . This system comprises of a transmis- 
sion-side cellular phone 20A and a reception-side cel- 
lular phone 30A connected via the mail server 40. 
[0033] In the system shown in Fig. 3, the reception- 
side cellular phone 30 decides the expression assign- 
ee) ment information itself based on the number of calls that 
have not been attended. On the other hand, in the sys- 
tem shown in Fig. 11, the transmission-side cellular 
phone 20A sends the expression assignment informa- 
tion to the reception-side cellular phone 30A. 
25 [0034] The transmission-side cellular phone 20A 
comprises of the portrait preparing section 21 that pre- 
pares a portrait and its attribute information based on 
the face image photographed by the camera 10, a por- 
trait storing section 24 that stores the portrait and the 
30 attribute information, a mail preparing section 25 that 
prepares a mail and attaches the portrait and the at- 
tribute information to the email, a mail transmitting sec- 
tion 26 that transmits the email to the mail server 40, an 
expression assignment information obtaining section 27 
35 that obtains the expression assignment information 
through interaction with the user, and the calling section 
23 that sends a call signal (i.e., makes a telephone call) 
to the reception-side cellular phone 30A and embeds 
the expression assignment information into the call sig- 
40 nal. The attribute information includes the position of the 
total face, the positions of the face parts, the color of the 
total face, and the color of the hair, and the like. It is 
preferable that, when' preparing a portrait, the portrait 
preparing section 21 prepares a portrait of a predeter- 
45 mined size. 

[0035] The reception -side cellular phone 30A com- 
prises a mail receiving section 39 that receives a mail 
with a portrait attached to it from the mail server 40, the 
face part storing section 33 that stores the face parts of 
so the portrait prepared to change the expression of the 
portrait, the portrait changing section 36 that changes 
expressions of the portrait attached to the mail, the por- 
trait output section 37 that outputs the portrait to the dis- 
play 11 , and the call receiving section 38 that receives 
55 a call signal with the expression assignment information 
embedded in it from the transmission-side cellular 
phone 20A. The call receiving section 38 extracts the 
expression assignment information embedded in the 
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call signal. 

[0036] The functions of sections of the transmission- 
side cellular phone 20A and the reception-side cellular 
phone 30A may be realized by hardware or software (i. 
e. , computer programs) or both . When the functions are 5 
realized using software, a recording medium such as a 
semiconductor memory may be provided to store the 
software. The face part storing section 33 stores the in- 
formation shown in Fig. 5. 

[0037] Same reference numerals have been attached 10 
to those sections in Fig. 3 and Fig. 1 1 that perform same 
or similar functions or have same or similar configura- 
tions. 

[0038] The transmission-side cellular phone 20A per- 
forms the process shown in Fig. 1 2, while the reception- 15 
side cellular phone 30A performs the process shown in 
Fig. 13. 

[0039] As shown in Fig. 1 2, the mail preparing section 
25 of the transmission-side cellular phone 20A reads the 
portrait stored in the portrait storing section 24, prepares 20 
the email, and attaches the portrait to the email (step 

551) . Generally, a portrait of the owner of the transmis- 
sion-side cellular phone 20A is stored in the portrait stor- 
ing section 24. Next, the expression assignment infor- 
mation obtaining section 27 obtains expression assign- 25 
ment information by interaction with the user or the own- 
er of the transmission-side cellular phone 20A (step 

552) . 

[0040] Then, the mail transmitting section 26 trans- 
mits the email (with the portrait and the attribute infor- 30 
mation attached) to the mail server 40 (step SS3). The 
mail preparing section 25 may be made to embed the 
attribute information into the header of the email instead 
of attaching the attribute information. 

[0041] Next, the calling section 23 transmits a call sig- 35 
nal with the expression assignment information embed- 
ded to the reception -side cellular phone 30A (step SS4). 
The expression assignment information may be embed- 
ded into a header of the call signal. Thus, the reception- 
side cellular phone 30A comes to know that the trans- *o 
mission-side cellular phone 20A has sent a mail to it. 
[0042] Thus, the portrait and attribute information are 
transmitted from the transmission-side cellular phone 
20A to the reception-side cellular phone 30A in the form 
of an email. On the other hand, the expression assign- 
ment information is transmitted from the transmission- 
side cellular phone 20A to the reception-side cellular 
phone 30A through the call signal. 
[0043] When a call signal is received by the call re- 
ceiving section 38 of the reception-side cellular phone so 
30A, the call receiving section 38 extracts the expres- 
sion assignment information embedded in the call signal 
(step SS5 in Fig. 1 3). The mail receiving section 39 re- 
ceives a mail from the mail server 40 and sends the por- 
trait and the attribute information attached to the email 55 
to the portrait changing section 36 (step SS6). 
[0044] Then, the portrait changing section 36 obtains, 
from the face parts storing section 33, the face parts with 



expressions relating to the expression assignment in- 
formation (step SS7). Then the portrait changing section 

36 changes the portrait obtained via the email by com- 
bining the face parts with expressions with the portrait 
(step ST8). Finally, the portrait output section 37 outputs 
the changed portrait to the display 11 where the portrait 
is displayed (step SS9). 

[0045] As a result, for example, if the caller is in happy 
mood, he can select expression assignment information 
that will display a face image that is smiling on the dis- 
play of the cellular phone of the receiver. Thus, the re- 
ceiver will know the feelings of the caller which realizes 
advanced communications between the users. 
[0046] It has been mentioned above to embed the ex- 
pression assignment information into the call signal, 
however, the expression assignment information may 
be attached to an email. Moreover, the expression as- 
signment information may be embedded into a header 
of the email along with the attribute information. 
[0047] Fig. 1 4 shows still another example of a system 
for realizing communications using the cellular phone 
shown in Fig. 1. This system comprises of a transmis- 
sion-side cellular phone 20B and a reception-side cel- 
lular phone 30B connected via the mail server 40. 
[0048] In the system shown in Fig. 11 , the transmis- 
sion-side cellular phone 20A sends the expression as- 
signment information to the reception -side cellular 
phone 30A and the reception-side cellular phone 30A 
changes the portrait of based on the expression assign- 
ment information. On the other hand, in the system 
shown in Fig. 14, the transmission-side cellular phone 
20B changes the portrait based on the expression as- 
signment information and sends the changed portrait to 
the reception -side cellular phone 30B for displaying. 
[0049] The transmission-side cellular phone 20B 
comprises of the portrait preparing section 21 that pre- 
pares a portrait and its attribute information based on 
the face image photographed by the camera 1 0, the por- 
trait storing section 24 that stores the portrait and the 
attribute information, a face part storing section 28 that 
stores the face parts of the portrait prepared to change 
the expression of the portrait, the expression assign- 
ment information obtaining section 27 that obtains the 
expression assignment information through interaction 
with the user, a portrait changing section 29 that chang- 
es the portrait, the mail preparing section 25 that pre- 
pares an email and attaches the portrait to the email, 
the mail transmitting section 26 that transmits the email 
to the reception -side cellular phone 30B via the mail 
server 40, and the calling section 23 that makes a tele- 
phone call to the reception-side cellular phone 30B. 
[0050] The reception -side cellular phone 30B com- 
prises of the mail receiving section 39 that receives the 
email from the mail server 40, the portrait output section 

37 that outputs the portrait to the display 11 , and the call 
receiving section 38 that receives a notification from the 
transmission-side cellular phone 20B that an email has 
been send to the reception-side cellular phone 30B. 
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[0051] The functions of sections of the transmission- 
side cellular phone 20B and the reception-side cellular 
phone 30B may be realized by hardware or software (i. 
e., computer programs) or both. When the functions are 
realized using software, a recording medium such as a 
semiconductor memory may be provided to store the 
software. The face part storing section 28 stores the in- 
formation shown in Fig. 5. 

[0052] Same reference numerals have been attached 
to those sections in Fig. 3, Fig. 11, and Fig. 14 that per- 
form same or similar functions or have same or similar 
configurations. 

[0053] The transmission-side cellular phone 20B per- 
forms the process shown in Fig. 15. 
[0054] The portrait changing section 29 reads the por- 
trait stored in the portrait storing section 24 (step SK1 ). 
Generally, a portrait of the owner of the transmission- 
side cellular phone 20B is stored in the portrait storing 
section 24. Next, the expression assignment informa- 
tion obtaining section 27 obtains the expression assign- 
ment information by interaction with the user or the own- 
er of the transmission-side cellular phone 20B and sup- 
plies the expression assignment information to the por- 
trait changing section 29 (step SK2). 
[0055] Then, the portrait changing section 29 obtains, 
from the face part storing section 28, the face parts re- 
lating to the expression assignment information (step 
SK3). Next, the portrait changing section 29 changes 
the portrait by combining the face parts (step SK4). 
[0056] Next, the portrait changing section 29 supplies 
the portrait to the mail preparing section 25, and the mail 
preparing section 25 prepares an email and attaches the 
portrait to the email (step SK5). The mail transmitting 
section 26 transmits the email with the portrait attached 
to it to the mail server 40 (step SK6). Finally, the calling 
section 23 sends a call signal to the receptionrside cel- 
lular phone 30B to inform the reception-side cellular 
phone 30B that an email has been sent (step SK7). 
[0057] The mail receiving section of the reception- 
side cellular phone 30B receives the email with the por- 
trait attached to it from the mail server 40, extracts the 
portrait from the email and supplies the portrait to the 
portrait output section 37. Finally, the portrait output sec- 
tion 37 outputs the portrait to the display 11 where the 
portrait is displayed. 

[0058] As a result, for example, if the caller is in happy 
mood, he can select expression assignment information 
so that a face image that is smiling is sent to the receiver 
as a mail attachment and displayed on the display of the 
cellular phone of the receiver. Thus, the receiver will 
know the feelings of the caller which realizes advanced 
communications between the users. 
[0059] Fig. 1 6 shows still another example of a system 
for realizing communications using the cellular phone 
shown in Fig. 1 . This system comprises of a transmis- 
sion-side cellular phone 20C and a reception-side cel- 
lular phone 30C connected via the mail server 40. 
[0060] In the system shown in Fig. 1 6, the transmis- 



sion-side cellular phone 20C sends face parts relating 
to desired expression assignment information to the re- 
ception-side cellular phone 30C via an email and the re- 
ception-side cellular phone 30C changes the portrait of 
5 based on the expression assignment information and 
displays the portrait. 

[0061] The transmission-side cellular phone 20C 
comprises of the face part storing section 28 that stores 
the face parts of the portrait prepared to change the ex- 
10 pression of the portrait, the expression assignment in- 
formation obtaining section 27 that obtains the expres- 
sion assignment information through interaction with the 
user, the mail preparing section 25 that prepares an 
email and attaches the face parts to the email, the mail 
15 transmitting section 26 that transmits the email to the 
reception-side cellular phone 30C via the mail server 40 , 
and the calling section 23 that makes a telephone call 
to the reception-side cellular phone 30C. 
[0062] The reception-side cellular phone 30C com- 
20 prises of the mail receiving section 39 that receives the 
email from the mail server 40, the portrait storing section 
32 that stores the portrait and corresponding attribute 
information downloaded from the transmission-side cel- 
lular phone 20C via the mail server 40, the portrait 
25 changing section 36 that changes the portrait, the por- 
trait output section 37 that outputs the portrait to the dis- 
play 11 , and the call receiving section 38 that receives 
a notification from the transmission-side cellular phone 
20C that an email has been send to the reception-side 
30 cellular phone 30C. 

[0063] The functions of sections of the transmission- 
side cellular phone 20C and the reception-side cellular 
phone 30C may be realized by hardware or software (i. 
e., computer programs) or both. When the functions are 
35 realized using software, a recording medium such as a 
semiconductor memory may be provided to store the 
software. 

[0064] The face part storing section 28 stores the in- 
formation shown in Fig. 5. Moreover, the portrait storing 

40 section 32 stores the information shown in Fig. 4. 

[0065] Same reference numerals have been attached 
to those sections in Fig. 3, Fig. 11, Fig. 14, and Fig. 16 
that perform same or similar functions or have same or 
similar configurations. 

45 [0066] The transmission-side cellular phone 20C per- 
forms the process shown in Fig. 17, while the reception- 
side cellular phone 30C performs the process shown in 
Fig. 18. 

[0067] As shown in Fig. 17, the expression assign- 
so ment information obtaining section 27 obtains expres- 
sion assignment information by interaction with the user 
or the owner of the transmission-side cellu lar phone 20C 
(step SA1). Then, the expression assignment informa- 
tion obtaining section 27 obtains face parts, from the 
55 face part storing section 28, relating to the expression 
assignment information and supplies the face parts to 
the mail preparing section 25 (step SA2). The mail pre- 
paring section 25 prepares an email and attaches the 
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face parts to the email (step SA3). Next, the mail trans- 
mitting section 26 transmits the email to the reception- 
side cellular phone 30C via the mail server 40 (step 
SA4). Finally, the calling section 23 sends a call signal 
to the reception-side cellular phone 30C to inform the 
reception-side cellular phone 30C that an email has 
been sent (step SA5). 

[0068] The mail receiving section 39 of the reception- 
side cellular phone 30C receives the email with the face 
parts attached to it from the mail server 40 (step SA6) 
and extracts the face parts from the email (step SA7). 
Next, the portrait changing section 36 obtains the tele- 
phone number of the transmission-side cellular phone 
20C (step SA8). Next, the portrait changing section 36 
obtains, from the portrait storing section 32, the portrait 
and the attribute information relating to the telephone 
number of the transmission-side cellular phone 20C 
(step SA9). 

[0069] Then, the portrait changing section 36 chang- 
es the portrait by combining the face parts with expres- 
sions with the portrait stored in the portrait storing sec- 
tion 32 (step SA10). Finally, the portrait output section 
37 outputs the changed portrait to the display 1 1 where 
the portrait is displayed (step SA11). 
[0070] As a result, for example, if the caller is in happy 
mood, he can select expression assignment information 
so that face parts that will display a face image that is 
smiling is sent to the receiver as a mail attachment. 
Thus, the receiver will know the feelings of the caller 
which realizes advanced communications between the 
users. 

[0071] Fig. 19 shows a structure of a cellular phone 
99, as an example of an information device, according 
to another embodiment of the present invention. 
[0072] The cellular phone 99 comprises of the camera 
10, the display 11, a portrait preparing section 100 that 
prepares a portrait and attribute information of the por- 
trait based on the face image photographed by the cam- 
era 1 0, a portrait storing section 1 01 that stores the por- 
trait and the attribute information, a portrait output sec- 
tion 102 that outputs the portrait stored in the portrait 
storing section 101 or the portrait prepared by the por- 
trait preparing section 100 (hereinafter "two portraits") 
to the display 11 , a portrait combining section 103 that 
combines the two portraits, and a relationship estimator 
1 04 that estimates the relationship between the persons 
of the portraits based on the two portraits. 
[0073] The functions of sections of the cellular phone 
99 may be realized by hardware or software (i.e., com- 
puter programs) or both. When the functions are real- 
ized using software, a recording medium such as a sem- 
iconductor memory may be provided to store the soft- 
ware. 

[0074] Fig. 20 shows an example of processes per- 
formed by the portrait combining section 1 03. This proc- 
ess starts when the user or the owner of the cellular 
phone 99 issues a command to start the process. 
[0075] The portrait combining section 1 03 first selects 
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two portraits to be combined, through interaction with 
the user (step SB1)! The portrait combining section 103 
selects two portraits to be combined from among the 
portraits prepared by the portrait preparing section 100 
5 and/or the portraits stored in the portrait storing section 
101. 

[0076] Next, the portrait combining section 103 se- 
lects a method for combining (hereinafter "combining 
method") the portraits through interaction with the user. 
10 Following methods may be considered as the methods 
for combining the two portraits A and B: 

(1 ) use the eyes and the mouth of the portrait A and 
use the total face and other face parts of the portrait 

is B (hereinafter "method M1 "), 

(2) based on random numbers determine which 
parts of the portrait A and B are to be used (herein- 
after "method M2"), 

(3) use the position information of the total face and 
20 the face parts of the portrait A, use the shapes of 

the total face and the face parts of the portrait B 
(hereinafter "method M3"), 

(4) use an intermediate value of the face parts of 
the portrait A and B (hereinafter "method M4"). 

25 

[0077] The portrait combining section 1 03 selects one 
of these combining methods through an interaction with 
the user. It should be noted that a combining method 
other than the combining methods mentioned above 

30 may be used. 

[0078] Next, the portrait combining section 103 de- 
cides whether the method M1 has been selected (step 
SB3). If it is decided that the method M1 has been se- 
lected, the portrait combining section 103 determines 

35 from which portrait the total face is to be used and from 
which portrait the face parts are to be used, through in- 
teraction with the user (step SB4). 
[0079] If the method M1 has not been selected at the 
step SB3, then it is decided whether the method M2 has 

40 been selected (step SB5) . If it is decided that the method 
M2 has been selected, the portrait combining section 
103 determines from which portrait the total face is to 
be used, and from which portrait each face part is to be 
used, by generating 1 or 2 at random (step SB6). 

45 [0080] If the method M2 has not been selected at the 
step SB5, then it is decided whether the method M3 has 
been selected (step SB7). If it is decided that the method 
M3 has been selected, the portrait combining section 
1 03 determines from which portrait the position informa- 

so tion of the total face and the face parts is to be used, 
and from which portrait the shape information of the total 
face and the face parts is to be used, through interaction 
with the user (step SB8). 

[0081] If the method M3 has not been selected at the 
55 step SB7, then it will mean that selected method is M4. 
In this case, the portrait combining section 103 calcu- 
lates the intermediate value of the two shapes and the 
positions respectively for each of the total face and each 
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face part, thereby to determine the total face and each 
face part to be pasted to form the composite portrait 
(step SB9). 

[0082] Finally, the portrait combining section 103 
combines the parts selected at step SB4 or SB6 or SB8 
or SB9 to obtain a changed portrait (step SB10). The 
changed portrait may be displayed on the display 11 or 
it may be transmitted to other person. 
[0083] It is mentioned above to combine parts of two 
portraits to obtain a changed portrait but parts of more 
than two portraits may be combined to obtain two or 
more portraits. 

[0084] Thus, according to the process shown in the 
flowchart of Fig. 20, parts of two different portraits can 
be combined as desired by the user. As a result, if, for 
example, portraits of a man and a woman (say husband 
and wife) may be combined to guess a face of would be 
child of the two. With this arrangement, it is possible to 
realize advanced communications between the users. 
[0085] Fig. 21 shows an example of processes per- 
formed by the relationship estimator 104. This process 
starts when the user or the owner of the cellular phone 
99 issues a command to start the process. 
[0086] The relationship estimator 1 04 selects two por- 
traits through interaction with the user (step SC 1 ). In oth- 
er words, the relationship estimator 1 04 selects two por- 
traits to carry out the estimation from among the portraits 
prepared by the portrait preparing section 100 and the 
portraits stored in the portrait storing section 1 01 . 
[0087] Next, the relationship estimator 104 obtains 
the attribute information of the portraits (i.e., the position 
of the total face, the positions of the face parts, the color 
of the total face, and the color of the hair) from the por- 
trait preparing section 100 and the portrait storing sec- 
tion 101 (step SC2). 

[0088] Next, the relationship estimator 1 04 calculates 
respective feature values of the portraits based on the 
attribute information of the portraits (where what kinds 
of feature values are to be used are set in advance), and 
calculates values of the parameters for estimating the 
relationship defined by difference values and ratios of 
the calculated feature values (step SC3). 
[0089] For example, the relationship estimator 104 
calculates feature values such as a distance between 
the two eyes, a size of a triangle formed by the two eyes 
and the mouth, a distance between a straight line 
formed by combining the two ears and a straight line 
formed by combining the two eyes, sizes of the eyes, a 
ratio of the size of the eye to the size of the ear, and the 
like. Then, the relationship estimator 104 calculates val- 
ues of the parameters for estimating the relationship de- 
fined by the difference values and the ratios of the cal- 
culated feature values. 

[0090] Next, the relationship estimator 104 substi- 
tutes the calculated parameters into the similarity calcu- 
lation expression prepared in advance, thereby to cal- 
culate the similarity between the two portraits selected 
to estimate the relationship (step SC4). 



[0091] Next, the relationship estimator 104 specifies 
the relationship corresponding to the calculated similar- 
ity. For example, when the relationship is within a certain 
range of similarity, the two persons of the portraits se- 
5 lected to carry out the estimation are defined such that 
the portraits represent the likeness between the parent 
(s) and the child, represent the likeness between the 
brothers, represent the likeness between the cousins, 
or represent no likeness at all. Based on this, the rela- 
te tionship estimator 104 specifies the relationship be- 
tween the two persons corresponding to the calculated 
similarity. 

[0092] Finally, the relationship estimator 104 makes 
the display 11 display the specified relationship. 
15 [0093] According to the flowchart shown in Fig. 21, the 
user can determine, for example, similarity between the 
portraits of two persons. With this arrangement, it is pos- 
sible to realize advanced communications between the 
users. 

20 [0094] Instead of outputting the relationship between 
the two persons in the expression form of the likeness 
between the pare nt(s) and the child, it is also possible 
to output the calculated similarity itself. 
[0095] Although a cellular phone is taken as an exam- 

25 pie of the information device, the present invention may 
be realized on any other information device such as a 
portable computer or a personal digital assistant (PDA). 
[0096] As explained above, according to the portable 
electronic device of the present invention, it is possible 

30 to change the expression of the portrait by using the por- 
trait having a reduced quantity of data generated based 
on the face image. With this arrangement, it is possible 
to realize advanced communications between the us- 
ers. 

35 [0097] According to the portable electronic device of 
the present invention, it is possible to generate, for ex- 
ample, the face of a child born between Mr. A and Ms. 
B by using the portraits having a reduced quantity of da- 
ta generated based on the face images. With this ar- 

40 rangement, it is possible to realize advanced communi- 
cations between the users. 

[0098] According to the portable electronic device of 
the present invention, it is possible to estimate, for ex- 
ample, a relationship between Mr. A and Ms. B by using 
45 the portraits having a reduced quantity of data generat- 
ed based on the face images. With this arrangement, it 
is possible to realize advanced communications be- 
tween the users. 

[0099] Although the invention has been described 
50 with respect to a specific embodiment for a complete 
and clear disclosure, the appended claims are not to be 
thus limited but are to be construed as embodying all 
modifications and alternative constructions that may oc- 
cur to one skilled in the art which fairly fall within the 
55 basic teaching herein set forth. 
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Claims 

1. An information device comprising: 

an image obtaining unit that is arranged to ob- 5 
tain a face image and a plurality of face part 
images corresponding to facial expressions; 
an expression determining unit that is arranged 
to determine a facial expression; and 
an image combining unit that is arranged to 
combine the face image and the face part im- 
ages corresponding to the facial expression to 
obtain a combined face image. 

2. The information device according to claim 1 , further 
comprising a communications unit for transmitting 
the face image, the face part images, or the com- 
bined face image to another information device. 

3. The information device according to claim 1 or 2, 
further comprising a digital camera for acquiring the 
face image and the face part images, wherein the 
image obtaining unit is arranged to obtain the face 
image and the face part images acquired by the dig- 
ital camera. 

4. The information device according to any of the pre- 
ceding claims, further comprising a memory for 
storing the face image and the face part images ac- 
quired via a communications line, wherein the im- 
age obtaining unit is arranged to obtain the face im- 
age and the face part images from the memory. 

5. The information device according to any of the pre- 
ceding claims, further comprising: 

a digital camera for acquiring the face image; 
and 

a memory for storing the face part images ac- 
quired via a communications line, 

wherein the image obtaining unit is arranged 
to obtain the face image from the digital camera and 
the face part images from the memory. 

6. The information device according to any of claims 
1 to 4, further comprising: 

a digital camera for acquiring the face part im- 
ages; and 

a memory for storing the face irpage acquired 
via a communications line, 

wherein the image obtaining unit is arranged 
to obtain the face part image from the digital camera 
and the face image from the memory. 

7. The information device according to any of the pre- 



ceding claims, wherein the face part images are im- 
ages of eyes, nose, and mouth. 

8. The information device according to any of the pre- 
ceding claims, further comprising an operating unit 
with which an operatorcan select one facial expres- 
sion from among a plurality of facial expressions, 
wherein the expression determining unit is ar- 
ranged to determine the selected facial expression 
as the facial expression. 

9. The information device according to any of the pre- 
ceding claims, further comprising: 

a receiving unit for receiving a telephone call; 
a number obtaining unit for acquiring a tele- 
phone number of the telephone call; 
a memory for storing face images and face part 
images in correlation with telephone numbers; 
and 

a display unit for displaying the combined face 
image, 

wherein the expression determining unit is ar- 
ranged to determine the facial expression based on 
a predetermined condition, and 

the image combining unit is arranged to com- 
bine the face image and the face part images cor- 
responding to the facial expression and corre- 
sponding to the telephone number to obtain the 
combined face image. 

1 0. The information device according to claim 9, where- 
in the predetermined condition is a number of tele- 
phone calls that were not attended in a predeter- 
mined period. 

i 

11. The information device according to claim 10, 
wherein the predetermined period is two hours. 

12. The information device according to any of the pre- 
ceding claims, further comprising a display unit for 
displaying the face image, the face part images, or 
the combined face image. 

13. An information device comprising: 

an image obtaining unit that is arranged to ob- 
tain a first face image and a second face image; 
a method determining unit that is arranged to 
determine a method for combining; and 
an image combining unit that is arranged to 
combine the first face image and the second 
face image based on the thus determined 
method for combining to obtain a combined 
face image. 

14. The information device according to claim 13, fur- 
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ther comprising a communications unit for transmit- 
ting the first face image, the second face image, or 
the combined face image to another information de- 
vice. 

15. The information device according to claim 13 or 14, 
further comprising a digital camera for acquiring the 
first face image and the second face image, wherein 
the image obtaining unit is arranged to obtain the 
first face image and the second face image ac- 
quired by the digital camera. 

16. The information device according to any of claims 
13 to 15, further comprising a memory for storing 
the first face image and the second face image ac- 
quired via a communications line, wherein the im- 
age obtaining unit is arranged to obtain the first face 
image and the second face image from the memory. 

17. The information device according to any of claims 
13 to 16, further comprising: 

a digital camera for acquiring the first face im- 
age; and 

a memory for storing the second face image ac- 
quired via a communications line, 

wherein the image obtaining unit is arranged 
to obtain the first face image from the digital camera 
and the second face image from the memory. 

18. The information device according any of claims 13 
to 17, further comprising an operating unit with 
which an operator can select the first face image 
and the second face image from among a plurality 
of face images. 

19. The information device according to any of claims 
13 to 1 8, wherein the method for combining is one 
or more selected from the group consisting of 

using eyes and mouth of the first face image 
and using a total face and other face parts of the 
second face image, 

using parts of the first face image and the sec- 
ond face image based on generation of random 
numbers, 

using a position information of the total face 
and the face parts of the first face image, and using 
shapes of the total face and the face parts of the 
second face image, and 

using an intermediate value of the face parts 
of the first face image and the second face image. 

20. The information device according to any of claims 
13 to 19, further comprising a display unit for dis- 
playing the first face image, the second face image, 
or the combined face image. 



21. An information device comprising: 

an image obtaining unit that is arranged to ob- 
tain a first face image and a second face image; 

5 a determining unit that is arranged to determine 

a similarity between the first face image and the 
second face image and a relationship between 
the persons of the first face image and the sec- 
ond face image determined from the similarity; 

10 and 

a display unit for displaying the similarity, the 
relationship or both the similarity and the rela- 
tionship. 

15 22. A computer program containing instructions which, 
when executed on a computer, cause the computer 
to perform: 

acquiring a face image and a plurality of face 
20 part images corresponding to facial expres- 

sions; 

determining a facial expression; 
combining the face image and the face part im- 
ages corresponding to the facial expression to 
25 obtain a combined face image; and 

displaying the combined face image. 

23. A computer program containing instructions which, 
when executed on a computer, cause the computer 
30 to perform: 

acquiring a first face image and a second face 
Image; 

determining a method for combining; 
35 combining the first face image and the second 

face image based on the method for combining 
to obtain a combined face image; and 
displaying the combined face image. 

40 24. A computer program containing instructions which, 
when executed on a computer, cause the computer 
to perform: 

obtaining a first face image and a second face 

45 image; 

determining a similarity between the first face 
image and the second face image and a rela- 
tionship between the persons of the first face 
image and the second face image determined 

so from the similarity; and 

displaying the similarity, the relationship or both 
the similarity and the relationship. 

25. A computer readable storage medium having 
55 stored thereon a computer program according to 
any of claims 22 to 24. 
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